Recently users have come to demand an interactive user interface enabling them to operate the digital contents with their hands on any surface where the digital contents are displayed. Such a system can use a set of fix mounted cameras installed around the operation region for detecting the fingertips. However, the fingers can often be occluded (or obstructed) by hand pose or by other hands when multiple hands are involved in the operation. Although increasing the number of fix mounted cameras can ease this problem, this can increase the cost, computational burden and complexity of the system.